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ABSTRACT 

Data re-sampling methods such as delete-one jackknife, bootstrap or the sub¬ 
sample covariance are a common tool for estimating the covariance of large scale struc¬ 
ture probes. We investigate different implementations of these methods in the context 
of cosmic shear two-point statistics. Using log-normal simulations of the convergence 
field and the corresponding shear field we carry out realistic tests of internal covari¬ 
ance estimators. For a survey of ^ 5000 deg^ we find that jackknife, if implemented in 
what we call the galaxy-scheme, provides the most reliable covariance estimates. Boot¬ 
strap, in the common implementation of duplicating sub-regions of galaxies, strongly 
overestimates the statistical uncertainties. 

In a forecast for the complete 5-year DES survey we show that internally esti¬ 
mated covariance matrices can provide a large fraction of the true uncertainties on 
cosmological parameters in a 2D cosmic shear analysis. The volume inside contours 
of constant likelihood in the flm-cs plane as measured with internally estimated co- 
variance matrices is on average > 85% of the volume derived from the true covariance 
matrix. The uncertainty on the parameter combination Eg ^ derived from 

internally estimated covariances is ^ 90% of the true uncertainty. 

Key words: large scale structure - cosmic shear - covariance - jackknife - angular 
correlation function 


1 INTRODUCTION 


Two-point statistics of cosmological random fields such as 
the cosmic shear correlation functions or the galaxy cluster¬ 
ing angular correlation function are common probes of the 
large scale structure of the universe. Recent m easurements of 
these co rrelation function s are e. g. reported in ThomasetajJ 


ll201ll): Kilbinge r et al.l (l2013h : Ide Simoni et al.l 1 20131) : 


iBecker et al.l In order to use these statistics for con¬ 

straining cosmological models one needs a quantitative de¬ 
scription of the joint distribution of the correlation func¬ 
tion estimators. When assuming multivariate Gaussian er¬ 
rors, this is given by the covariance matrix. On large an¬ 
gular scales this covariance matrix can - both for cosmic 
shear and galaxy clustering - be well descr ibed by a Gaus¬ 
sian approximation for the involved fields JSchneider et al.l 
I2OO2I : [Crocce et^l 201 ll ). It has, however, been shown, that 


* E-mail: oliverf@usm.uni-muenchen.de 


the Gaussian approximation fails to de scribe the true PDF 
of the weak lensing convergence field (|Taruva et al.l 120021 : 
IVale &: WhitdFoOsI) and that it underestimates the true co- 
variance of the cosmic shear correlation functions on small 
scales, which can be alleviated by an e mpirical re-scaling 
dSemboloni et al] 2007: Sato et ^|2011^ . a log-normal ap- 
prox imation Jll ilbert et~ ■II2011I). or halo mode l approaches 
(e.g. iGoorav fc HiJ 200 ll : iTakada fc JainI l2009l : lEifler et"^ 
l2014h . 


Alternatives to modelling the covariance matrix are to 
estimate it from many independent realisations of cosmo¬ 
logical N-body simulations or to estimate it internally, i.e. 
from the data itself. The latter method is independent of 
assuming a particular cosmological model and is hence of¬ 
ten used to complement the other methods dKilbinger et al.l 
I2OI3I : IWang et al.ll2013l : iBecker et aLll2015l .). 


So far the performance of internal covariance estimators 
has only been systematically studie d for the galaxy clus ter- 
ing 2-pt function (in most detail bv iNorberg et al]l2009l ) or 
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for cross-correlations of the Cosmic Microwave Background 
(CMB) and the galaxy field dCabre et al.ll200^ '). In our pa¬ 
per, we will concentrate on cosmic shear correlation func¬ 
tions. We will show that the shape noise part of the covari¬ 
ance can be very accurately estimated internally while the 
cosmic variance part is generally underestimated. Gaussian 
simulations of the convergence field hence yield an overly 
optimistic test of internal covariance estimators, since the 
Gaussian model underpredicts the cosmic variance contri¬ 
bution to the covariance. We overcome this problem by em¬ 
ploying log-normal simulations of the convergence field. 

In our paper we want to study the performance of in¬ 
ternal covariance estimators such as bootstrap, jackknife 
or the sub-sample covariance. There is no complete agree¬ 
ment in the literature yet on whether internal covariance 
estimates can be used to constrain cosmological parame¬ 
ters from measured 2pt-correlations or whether they are a 


mere tool to generate reasonable errorbars in 

slots of cor- 

relation functions (see e.g. 

Norberg et al 

120091: 

Wang et al.l 

l2013l:lde Simoni et al.ll2012 

; Tavlor et al.l 

201't. 

We want to 


address the questions of how many internal resamplings are 
required in order to get a stable covariance matrix, whether 
internal estimators over- or underestimate the covariance 
matrix and whether/how internal covariance estimates can 
yield unbiased estimates of the inverse covariance matrix. 

Our paper is organized as follows: In section [2] we in¬ 
troduce the cosmic shear correlation functions and explain 
the Gaussian and the log-normal model for the covariance 
of 2-pt. function estimators. In section [3] we describe the 
simulations we use to generate mock shape catalogues that 
follow any given input power spectrum and whose underly¬ 
ing convergence field has a log-normal PDF. These are the 
simulations with which we will test the performance of in¬ 
ternal covariance estimators. 

In section |4] we introduce two distinct ways of perform¬ 
ing jackknife estimation of the covariance of two-point mea¬ 
sures - the pair-jackknife and the galaxy-jackknife. Further¬ 
more, we are explaining why jackknife, bootstrap and sub¬ 
sample covariance are almost equivalent. 

In section [S] we apply internal covariance estimators to 
simulated cosmic shear surveys. We show that in the pair- 
scheme all estimators are almost identical and we demon¬ 
strate the systematic effects of the different estimation 
schemes when varying the number of re-samplings. Our 
method to find optimal estimation schemes has to be re¬ 
run for any specific survey, because the performance of 
internal estimators depends crucially on the depth and 
area of a survey. In the end of section [5] we configure our 


simulations to match the complete, 5-year Dark Energy 

Survey (DES, 

The Dark Energy Survey GollaborationI 

2003: 

lFlaugherj|2005 

) and test the accuracy of jackknife covariance 


matrices for this particular setting. The code used for our 
simulations is made publicly availablf0- 

In section [6] we discuss the results of our work. 


2 COSMIC SHEAR BASICS 


2.1 Cosmic Shear Correlation Functions 

Cosmic shear measures the correlated distortion of galaxy 
shapes due to gravitational leasing by the large scale struc¬ 
ture of the universe as a function of the angular dis¬ 
tance of galaxy pairs on the sk y. We follow here the no¬ 
tation of ISchneider et al.l ll2002l l and employ the flat-sky- 
approximation, i.e. we assume a tangential Cartesian coor¬ 
dinate system i? = (i?i, i92) on the sky. 

In this coordinate system the comic shear field is at each 
point characterized by a complex number 7 ( 1 ?) = 71 -I- ^ 72 - 
If the separation vector Ai? = 'i?2 — of two points on 
the sky has the polar angle (j) then the tangential and cross 
components of 7 at 'i?2 and iJi (with respect to each other) 
are defined as 

7 t = -Tie ; 7x = -Tm (je~‘^'''^^ . (1) 

The cosmic shear correlation functions ^±{0) are defined as 
the expectation values 

C±(^) = (771772) ± (7x,i7x,2) , (2 ) 

where 6 is the absolute value of Ai?. It can be computed in 
terms of the power spectrum Pk{(.) of the scalar convergence 
field Kif)) as 

^±(0) = ^ ^ , (3) 

where Joix) {Ja{x)) is the 0-th order (4-th order) Bessel 
function. 

The shape of a galaxy can be characterized by a complex 
number e which is to first order the sum of the intrinsic shape 
e'" of the galaxy and the distortion caused by gravitational 
leasing, i.e. the value 7 ( 1 ?) at the location 1 ? of the galaxy, 

e = -I- 7 . (4) 


In a cosmic shear s urvey the shapes ej of many galaxies 
are measured and (cf. ISchneider et al.ll2002f l an estimator 
for the correlation function can be constructed as 


i±{o) 


J2ijWj'Wj{et,iet,j ±ex,iex,j)Afl(ij) 
Eii WjWjAeiij) 


(5) 


where we have allowed for some weighting scheme Wi for the 
shape measurements and where the filter Ae{ij) selects all 
galaxy pairs {i, j) in the survey whose angular separation lies 
in some finite bin around 0. The normalization in equation 
[S]is the effective number of galaxy pairs in a bin around 9, 
which we will abbreviate as 


Npi9) = Y2 ‘Wj‘^0■ ( 6 ) 


1 www.usm.uni-muenchen.de/people/oliverf/, the code also con¬ 
tains many other useful features, that e.g. enable the user to cre¬ 
ate mock data suitable for galaxy-galaxy lensing or galaxy clus¬ 
tering measurements. 


2.2 Covariance of the Correlation Functions 

The covariance matrix of the estimator in equation [5] is de¬ 
fined as 

C±,±(6ii,6i2) = - i±{9i)){i±{92) - i±{ 62 ))) 

= {i±{9i)i±(e2)) - i±{9i)i±(92) ■ ( 7 ) 
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In order to compute this covariance matrix it is convenient 
to split ^±{0) into the three different contribution 


ine) 

i±{o) 

iTC) 


N,{e) 

T,ij ±'fx,iyx,j)Ae(ij) 

J2ij WiwEefilt,] ± e'x.ilx j)Ae(ij) 


( 8 ) 


which are the autocorrelation of the intrinsic shape noise, 
the autocorrelation of the shear signal and their cross cor¬ 
relation. The whole estimator [5] is given in terms of these 
as 


ipe) = iT{e) + CP {9) + 2 . |r( 0 ). 


Under the assumption that the shear signal and the shape 
noise are independent of each other it is obvious that 

(cr («i)cr( 02 )> = 0 = {c± (eoff ( 02 )). 

If the intrinsic shape of any two galaxies is assumed to be 
uncorrelated, we can also conclude that 

iiT) = 0 for 0 > 0 (9) 

and hence 

(cr(0i)ff(02)) = (cr(0i)} ■ (I? (02)) = 0 for 01,02 > 0. 

The covariance matrix can thus be split into three different 
contributions, 

C7±.± = UJE + UiE + C7E± , (10) 

namely 

ur±(0i,02) = (cr(0i)lr(02)), 

C7iE(ei,e2) = (ff(0l)ff(02))-a(0l)a(02) , 

ci"±(0i,02) = 4.(i3i‘(0i)er(e2)). (11) 

The term depends on 4-point functions of the shear 

field and is called the cosmic variance term. In order to 
evaluate it, further assumptions on the probability distribu¬ 
tion function (PDF) of the shear or the convergence field 
are needed and we will discuss two possible models for the 
convergence PDF in sections 12.2.II and 12.2.21 - the Gaussian 
and the log-normal model. 

The contributions a-nd C+i+ can be computed 

without additional assumptions. In Ijoachimi et al.l feoOSl i 
it is derived that they are given b£l 

^^±( 01 , 02 ) = Jo/Cm) Jc/ApQi) PCP . 

C?![(0i,02) = G!!T(0i,02) 



C;!^(0i,02) = 0, (12) 

where A is the survey area, fi is the number density of galax¬ 
ies, (Te is the dispersion of the intrinsic ellipticity which is 
defined as 

2 / in in*\ /-i o\ 

:= (e e ) , (13) 


^ as in ISchneider et ^ ||2002| 1 they employ an ensemble average 
over the galaxy positions to derive their expressions. 


and Pk is again the convergence power spectrum. 

2.2.1 Gaussian Approximation 

In the paper series by Schneider et ahl (l2002il . 
iKilbinger fc Schneider! (|2004lf and Joachimi et al.l (l2008l 'l the 
covariance matrix is studied in the Gaussian approximation, 
i.e. assuming that the convergence field has a Gaussian 
PDF such that its 4-point correlation functions can be 
expressed in terms of its 2-point correlation functions. 

For the case where the survey geometry is much larger 
than the angular sca l es con sidered in the correlation func¬ 
tions, [joachimreF^^ ll2008li derive the following expressions 
for the cosmic variance term: 

C7i“±(0i,02) = ^f dllJo/Piep Jo/4(^02) Pip). (14) 

However, due to the finite geometry of any given survey 
equation 1141 generally overestima tes the covariance of Gaus¬ 
sian field as was demonstrat ed inISato et al.1 (1201 ll i. This fi¬ 
nite area effect according to lSato et al.1 is not important for 
surveys larger than 1000 deg^. For smaller su rveys a method 
developed in IKilbinger fc Schneider! ll2004h which doesn’t 
employ an ensemble average over galaxy positions should 
be used to evaluate the Gaussian covariance. This method 
was for example used in the analysis of CHFTLenS data in 
IKilbinger et al.1 ll2013ll . The finite area effect is also impor¬ 
tant for internal covariance estimation and will be further 
discussed in section 


2.2.2 Shifted Log-Normal Approximation 

As e.g. rep orted by Ta ruya et af] ll2002h . IVale fc White! 
1 I 2 OO 3 II or bv iHilbert et al.| ||2011^ Hie Gaussian model fails 
to describe the true PDF of the convergence and especially 
on small separations poorly represents the true covariance 
of t he cosmic shea r 2-po int functions. 

iHilbert et al.1 (l201lli propose a different model for the 
convergence PDF, namely that of a zero-mean shifted log¬ 
normal distribution. In this approach the convergence at a 
given point on the sky is assumed to be of the form 


k{6) = exp[n(0)] — kq (15) 

where n{G) is a Gaussian random field (not necessarily with 
a vanishing mean) and the mini mal convergen c e par ameter 
kq is chosen such that (k) = 0. iHilbert et al.l (1201 ih show 
that from the corresponding PDF a model for the shear- 
shear contribution to the covariance matrix can be derived. 
Considering only the most important terms they also pro¬ 
vide a simplified log-normal covariance, which reads 

Ci'’± (01,02) = ^Jd££ Jo /A {£9a ) Jo/4 {£92) pI {£) 

+ -T7 C{ei)C{e2) d0 0 C+(0) ,(16) 

^ 0 ^ 40 

where 9a represents the ’radius’ of the survey, given by 



Comparing equation 1161 to equation 1141 on can see that the 
simplified log-normal approximation to CKj- consists of only 
one correction term to the Gaussian model. In our paper, we 
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will simulate log-normally distributed convergence fields and 
use equation US] to compute the cosmic variance part of our 
model covariance. 


2.2.3 Finite bin width 

The expressions presented above for the covariance of 
are derived under th e assumption of small angular bins 
llSchneider et al.ll2002ll . However, in section I5.2l we need cor¬ 
rect covariance expressions also for data vectors where the 
relative bin width is ~ 0.3, i.e. where the assumption of small 
bins does not hold. This is in fact the more realistic case, 
since broad bins are c ommonly used to redu c e the number 
of dat a points (see e.g. iKilbinger et al.l ll2013ll . feecker et al.l 

dioIslH . 

Hence, in section we proceed as follows: We first 
compute the log-normal model for the covariance, eqn. 1161 
for a set of very small angular bins 8i,i = 1,... ,N. Then 
we apply a linear transformation that takes the large data 
vector of the small angular bins to a smaller data vector by 
putting together p neighbouring bins of the old data vector, 

Oj = E 

*=P' 0 - 1 ) + 1 i=P' 0 '-l) + l 

iiOj) = E E 

*=p'(t-l)+l *=P'(J- 1)+1 

(18) 

where Np{0i) is the number of pairs in the ith bin of the 
finer data vector. 

The same linear transformation is then applied to the 
covariance matrix of the large data vector to get the covari¬ 
ance matrix of the compressed data vector. We find that for 
this decreases the mixed- and cosmic variance part of 
the covariance by > 30%, while for it makes almost no 
difference. The reason is that adjacent bins in are much 
more correlated than adjacent bins In . Hence, if two bins 
in are joined, the variance of the joined bin is almost 
identical to that of the individual bins and eqn. (TH] can still 
be appliecj^. 


3 LOG-NORMAL SIMULATIONS 

ISimon et al.l (l2004h describe a quick method to simulate cos¬ 
mic shear surveys based on a Gaussian convergence field for 
any given convergence-power-spectrum. On a quadratic grid 
in 2D-Fourier space they generate at each point £ of the grid 
a value of the convergence 

k{£.) = ki(£) -I- iK2{£.) 

where the components Ki{£) are drawn from a Gaussian dis¬ 
tribution with zero mean and variance 

4 - 

Here Pk is the desired convergence power-spectrum and V 


is the volume of the grid in angular space which is given in 
terms of the grid spacing Ai as 


V = 



(19) 


In order to achieve a convergence field that is real valued in 
angular space one has to impose the condition 


fi(£) = k*(-£) 


and in Fourier space the shear field is related to the conver¬ 
gence field by the equatioij^ 


£?-£l + 2i£i£2.,P, 

T'W =- 72 - 


( 20 ) 


A Fourier transform then gives the shear field in angular 
space. 

The main idea in generating a log-normal random field 
is to genera t e a G aussian field n(ff) with the method of 
ISimon et al.l ll2004h and transform it int o k(0) via equa¬ 
tion 1151 According to iMartin et al.l (l2012h ; iTakahashi et al.l 
1 I 2 OI 4 I 1 the power spectrum of n(0), Pn, can be computed 
from Pk as follows: 

relate d to the 2-pt. functi on of n{9) via (see e.g. equa¬ 
tion B.8 of lHilbert et al.ll201lll : 

First, the 2-pt. function of k,{9) is given in terms of the 
power spectrum P^ by 


G{G) = 1^ r d££ P4i) Mie). 

Jo 


Next, the 2-pt. function is relat ed to the 2-pt. func tion 
of n{9) via (see e.g. equation B.8 of lHilbert et aLlboili l 

^n(0) = ln(^4e)/«g + l) , 


where kq is the minimal convergence parameter from eqn. 
1151 Finally, the power spectrum of the Gaussian field n{9) 
by 

rao 

Pr^ii) = 2n dee en(0) Jo{£e). (21) 

Jo 

The fiel d n(9) can now be generated as described by 
ISimon et al.l (l2004|j . However, this way n{9) will have a 
mean value of zero. In order to ensure that (n) — 0 the 
mean value 

2 

p = Ko — (22) 

has to be added, where is the variance of the Gaussian 
field. The convergence field k.{9) now has to be transformed 
into Fourier space. Using equation [20] one can then compute 
the Fourier modes of the shear field and another Fourier 
transform gives the desired shear field in angular space. 


3.1 Setup and Validation of the Simulations 

The harmonic space grid we are using has a total number of 
(2^®)^ grid points and a grid spacing of A£ = 2. Hence, in 
each axis it ranges from —fmax = —2^® to fmax = -1-2^®. All 
modes 'y{£) with \£\ > fmax (i.e. the corners of the grid) are 
set to zero. The mode 7 g is also set to 0 and all other modes 


® a similar reasoning can be applied for the off-diagonal terms of 
the covariance 


^ see eqn. 2.1.11 of iKaiser &: Squirt ll 199311 or eqn. 25 of 
ISimon et al.l <l2004fl 
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are generated as explained above. Especially, we have to fix 
a cosmology and assume a certain redshift distribution of 
sources, p{z), to compute the convergence power spectrum 

Following eqn.[T9]the grid in angular space has a volume 
oi V = 2 -kI2 ~ lO^^deg^. Out of the center of that volume 
we will cut out a sub-grid of size A. Onto that sub-grid 
we are uniformly placing galaxies with a certain number 
density rigai. The shear of each individual galaxy is then 
determined by quadratic interpolation of the grid onto the 
galaxy position. Finally, a Gaussian intrinsic shape noise 
with an ellipticity dispersion Ue is added to get the total 
shape of the galaxy. Note that we simply added the shear 
signal and intrinsic ellipticity, hereby ignoring the effects of 
reduced shear. 

In this work we alwa ys keep the cosmology fixed to 
that of lHilbert et al.l (l201ll '). i.e. a flat ACDM universe with 
fib, ug, hioo, ris) = (0.25,0.045,0.9,0.73,1.0). To com- 
pute the converge nce power spectrum we employ halofit 
JSmith et al.ll200'3 i using the open source code nicae 43- The 
source distribution is taken to have the form 

^ e“(^) , where zo = ' (23) 

This i s the same form that was also assumed bv iHilbert et al.l 
ll201lli . The ellipticity dispersion is always set to 0.3 per 
component, i.e. = \/(2) • 0.3. All other quantities, i.e. 
area A, source density Wgai and median redshift Zmedian, will 
be varied throughout section[5] The different setups are sum¬ 
marized in table [T] 

The redshift d istribution of setup I is exactly that of 
iHilbert et al.l (I2OIII ) and imitates a rather deep survey com¬ 
parable e.g. to euclid. In this setup, we measure the 2-pt. 
correlation functions in 35 logarithmic bins from Omin = 
to @max = 150'. The area A was taken to be a square of 
70 deg X 70 deg. The minimal converg ence paramete r kq was 
chosen to be 0.032 as suggested by IHilbert et al.l for this 
redshift distribution. 

The area, galaxy density and redshift distribution of 
setup Ila are chosen to be similar to that of DES science 
verific ation data (DES-SV) which was used in iBecker et al.l 
ll2015h . In this setup, we measure the 2-pt. correlation func¬ 
tions in 15 logarithmic bins from Smin = 2' to 6'max = 300', 
which is also exactly the data vector used by iBecker et al.l 
1I2OI5II . In this setup we also reproduce the irregular shape 
of DES-SV, i.e. we use an SV-shaped healpix mask to cut 
out the sub-volume A. 

The setups Ilb and lie are aimed at a forecast for the 
final 5-year DES data. In lib we are assuming the same 
source density as in DES-SV and in lie a slightly higher 
one. Note that in principle, when adjusting the source den¬ 
sity, we should also adjust the source median redshift of 
the sources. But we will ignore this point, since our redshift 
distribution is anyway only a rough match to that of DES. 
Thus, for all setups Ila, lib and lie we take a median redshift 
of 0.7. Furthermore, for all these setups we use an empiri¬ 
cal formula ko{z) found by IHilbert et al.l (1201 il l to fix the 
minimal convergence parameter. Inserting the mean redshift 
of 2mean ~ 0.745 gives a value of kq = 0.019. The area in 
setups Hb and He are simply taken to be square shaped. 


setup 

A [deg] 

^gal 

^median 

KO 

1 

4900 

20 

1.0 

0.032 

Ila 

~ 150 

6 

0.7 

0.019 

lib 

5000 

6 

0.7 

0.019 

lie 

5000 

10 

0.7 

0.019 


Table 1. The different configurations of mock catalogs used in 
this paper. 



Figure 1. Comparison of the mean correlation functions from 
1000 simulations (red dots) and the input model (blue line). The 
the red error bars show the standard deviation of the mean and 
the green errorbars show the standard deviation of the single cor- 
re lation function meas urements. We used the redshift distribution 
of lHilbert et al.l ll2nilh to compute the input power spectrum and 
we also used their value of kq to generate the log-normal conver¬ 
gence. Note that in section f5.2l we will use a different configura¬ 
tion. 


® by Kilbinger et al., www.cosmostat.org/software/nicaea/ 
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To validate our simulations, we generate 1000 inde¬ 
pendent realizations of setup I. In order to speed up the 
computations we decrease the number of galaxies with re¬ 
spect to our jackknife analysis by a factor of 5, i.e. to 
JT'gai = 4/arcmin^, while at the same time decreasing the 
ellipticity dispersion by a factor of %/5- This way the covari¬ 
ance expressions in equation M stay unaffected. 

In figure[T]we show the mean measured correlation func¬ 
tions in the mock surveys. The correlation function measure¬ 
ment was carried out using the TreeCorr tree codfl The 
measured correlation functions and those derived from the 
input model agree well on most scales. Only at small angu¬ 
lar scales the measured value of differs significantly from 
the input model. The reason is the artificial cut-off at high 
values in our Fourier grid which both in the model and the 
simulation introduces artefacts - as can be seen from the os¬ 
cillatory behaviour of . To keep our analyses in section [5] 
free from these artefacts we will only consider those bins in 
that have 9 > 4.5'. For we continue to use a range of 
1' < 9 < 150'. Also, for the setups Ila to He (not shown 
here) the discrepancy in turns out to be less significant. 
Hence for these setups we stay with Smin = 2'. 

Figure [2] compares the sample covariance of the 1000 
simulations to the predictions from eauation llOl The relative 
deviation between measured variance and the log-normal 
model is ^ 20% for and ^ 15% for For both correla¬ 
tion functions these deviations seem to be significant given 
the uncertainties of the sample covariance estimate. How¬ 
ever, the sample variance values at different angular scales 
are highly correlated, which makes a ’x-by-eye’ judgement of 
the fit impossible. When transforming the covariance matri¬ 
ces into the eigenbasis of the model covariance (right-hand 
panel of figure [2]), the variance values become uncorrelated 
and the agreement of the covariance matrices becomes more 
evident. The eigenvalues at which the log-normal covariance 
significantly differs from the sample covariance of our sim¬ 
ulations are 3 orders of magnitude smaller than the biggest 
eigenvalues for 5+ and more than 2 orders of magnitude 
smaller than the biggest eigenvalues for (c.f. right-hand 
panel of figure [2]). Finally, our analyses in section [5] remain 
unchanged when the log-normal covariance is exchanged by 
the sample covariance of the 1000 independent realizations, 
which validates the simulations for our purposes (cf. ap¬ 
pendix]^ hgure . 


by Jarvis et al., github.com/rmjarvis/TreeCorr 
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Figure 2. Left: sample variance from 1000 independent simulations compared to the log-normal input model. The errorbars are assuming 
a Wishart distribution, note however that the different sample variance values are correlated. Right: in the diagonal basis of the model 
covariance matrix the sample variance values should independently follow a x^-distribution. The model and the simulations are consistent 
for the Ri 20 largest eigenvalues of the model covariance matrix. 


4 INTERNAL COVARIANCE ESTIMATION 
FOR TWO-POINT CORRELATION 
FUNCTIONS 

Suppose the correlation functions have been measured in 
finite bins around a set of angular distances 9i, i = 1,... ,d. 
Let 4 be either one of the data vectors [C±(^i), ■ • ■ ,C±(^d)] 
or the joint data vector of both correlation functions. 

If ^[7r] is a model for the measurement ^ which de¬ 
pends on a set of parameters tt, then a common statistic 
for constraining the possible values of tt is the statistic 


llKilbinger fc Schneiderj[200l L i.e. 

X^[7r] = (^-^[7r]fC-^ (^-^[Tr]) , (24) 

where C is the covariance matrix of 4- One way to get the co- 
variance matrix is to model it theoretically. As we have seen 
in section [22] the modelling of the covaria nce depends cru¬ 
cially on the PDF of the convergence field llSchneider et al.l 
I2OO2I : lililbert et al.ll20lil : ISato et al.ll20ll] 'l and neither the 
Gaussian nor the log-normal approximation match a realis¬ 
tic convergence PDF. Also, the model covariance matrix will 
depend on cosmological parameters itself which, at least for 
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small surveys, has to be taken into acco unt when deriving 
parameter constraints (lEifler et al.ll200^i . 

A way to get around modelling the covariance ma¬ 
trix directly is to use the sample covariance of measure¬ 
ments of the correlation functions in a set of independent 
N-body simulations (cf. iThhahashLgLaP |2009| ; ISato et ^ 
I 2 OO 9 I : lHarnois- Deraj3s_&^_y<M_^^rbel^ 20151 or for an ap¬ 
plication to data Kilbinger et al. 2013l l which however still 
depends on the model parameters, i.e. on the assumption 
of a particular cosmological model. Another alternative to 
modelling the covariance matrix is to estimate it from the 
data itself. In the following we will introduce three differ¬ 
ent internal covariance estimation methods - the sub-sample 
covariance, the del e te-one-jac kknife and the bootstrap (cf. 
iNorberg et al.|[200^ : lLohll200§ I. 


4.1 Subsample Covariance 

Let us split the area A of our cosmic shear survey into N 
equally shaped and sized subregions of the area As = A/N. 
In each subregion a = 1,..., N, a measurement of the data 
vector ^ can be carried out. Assuming that each sub-region 
has approximately the same number of galaxies and that the 
correlation functions are measured on scales much smaller 
than VAs the measurement of ^ in the whole survey is given 
by 

(25) 

q: = 1 

i.e. it is the mean values of the measurements in the sub- 
regions. If the measurements ^ are independent, then the 
ij-th element of their covariance matrix can be estimated by 

(A^A^J) « ^ - 0. if - 0. . (26) 

/ 3=1 

where A^ is the difference between ^ and its expectation 
value 

^=(r) = (4) • (27) 

Accordingly, if the assumption of independent sub-regions 
were true, the covariance of the total measurement ^ could 
be estimated by 

(28) 

'' ' a.= l 

We will c all the estimator in e quation [2^ the sub-sample co- 
variance (iNorberg et al.l[2009ll . The main systematic effects 
of internal covariance estimation can be most easily under¬ 
stood in terms of this estimator. Hence, before introducing 
the jackknife and bootstrap estimator, we will explain these 
systematics in the following two sections. 


4.2 Correlation of sub-samples 

The sub-sample covariance estimator relies on the assump¬ 
tion that the data is split into independent sub-samples, i.e. 
that there is no correlation of the measurements of the cor¬ 
relation functions in the different sub-regions, 

(Ai“A^^^> = 0 , for a ^ . (29) 


This can be seen from the fact that eqn. simply rescales 
the sub-field-to-sub-field covariance by a factor of 1/N to 
get the covariance of the whole survey. If the sub-samples 
are corr elated, this will underes timate the true covariance 
matrix llNordman fc Lahirill2007l i. 

Another way to think about this is as follows: the sub¬ 
sample covariance estimator assumes that the covariance 
matrix of ^ is inversely proportional to the survey area A. 
Hence it estimates the covariance of sub-regions of the size 
As within the data and then rescales it to the total area, 

C = E . Cs = 1 ■ Cs , (30) 


where N is again the number of sub-regions. But already 
from the log-normal model for the covariance it can be seen, 
that this rescaling is not valid. The log-normal correction 
term to the Gaussian covariance matrix is given by 

$±(^0^(02) Ae e c+(e). 

^ 0 ^ JO 


This term may be proportional to 1/A, but the upper inte¬ 
gral boundary also depends on the survey diameter Oa- As 
A increases, the covariance therefore decreases slower than 
1/A. Hence, assuming 1/A scaling when extrapolating from 
the covariance of the smaller sub-helds to the covariance of 
the full area underestimates the full covariance. Also, note 
that even the Gaussian covariance term in eqn. [12] is only 
an approximation for large survey sizes A. It also suffers 
fr om a finite area effect as can be seen from its derivation 
Schneider et al.l ll2002| j or its form given in [Hilbert et al.l 


1120111 ) . 


The fact that sub-samples should be as uncorrelated as 
possible is also the reason why the re-sampling of the data 
should be done into spatially connected patches. If instead 
the data would be randomly divided into sub-samples then 
the shear correlations in the sub-samples would be almost 
identical. Hence, only the shape-noise contributions to the 
covariance would be measured by such an estimator. 


4.3 Galaxy pairs crossing between sub-samples 

A problem specihc to the internal covariance estimation for 
two-point correlation functions is the question of what to do 
with pairs of galaxies where each galaxy lies in a different 
sub-region of the survey. 

In fact, the pieces of information in a cosmic shear sur¬ 
vey are not the individual galaxy shapes but the pairs of 
galaxy shapes. If the pairs crossing between sub-regions are 
completely ignored when computing the sub-measurements 
then one is re-sampling a data set that has less informa¬ 
tion than the total measurement of and hence a larger 
variance. Note, that this does not only influence the shape- 
noise part of the covariance but also the cosmic variance 
part. The reason is that galaxies at the edge of a sub-region 
contribute less terms to the correlation function measure¬ 
ment than galaxies in the center of the sub-region (c.f. 6g- 
ure 0, i.e. the area of the sub-patch is not uniformly probed 
by the galaxy pairs and the measured shear correlations are 
dominated by the inner part of the patch. In contrast to the 
correlation of sub-samples discussed before, this increases 
the cosmic variance between the sub-samples and can bias 
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Figure 3. Galaxies at the edge of a sub-region (in red) contribute 
less pairs to the measurement of the correlation functions (i.e. to 
equation [ 5 ] applied to the sub-sample) than galaxies in the center 
of the sub-region (in blue). Consequently, the area of the sub¬ 
patch is not uniformly probed by the galaxy pairs. This increases 
the cosmic variance between sub-regions and biases the covariance 
estimates high. Hence, it has an opposite effect to the correlation 
of sub-samples, which biases the covariance estimates low. As seen 
from the left-hand panel of figure [5] at large angular scales this 
can even lead to an overestimation of the cosmic variance of 
(in the galaxy-scheme, c.f. also figure [^. 




i 

/ 




y 

Ef / 




Figure 4. Two basic schemes for dividing a set of galaxy pairs 
into sub-samples: For each sub-region of the survey, there will 
be galaxy pairs crossing from that region into another (upper 
panel, green and red). In the galaxy-scheme is computed by 
considering only pairs that completely lay within the sub-region a 
(lower right panel). In the pair-jackknife scheme (lower left panel) 
half of the pairs that cross from a. to another region (drawn in 
green) are taken into account for computing while only the 
other half (red) is discarded. 


the covariance estimate high - especially on large angular 
scales. 

This effect can in principle be resolved by re-sampling 
the set of pairs (instead of the set of galaxy shapes), i.e. by 
defining the sub-measurement ^ as 

iw) = 




iexEx) + E 


half of cross pairs 


(4e^±4ei) 


Vo 


(31) 


How this resampling of galaxy pairs can be done is illus¬ 
trated in figure m Especially one has to make sure that each 

^ a 

galaxy pair enters exactly one of the ^ . We call this proce¬ 
dure the pair-scheme while we will call the standard proce¬ 
dure of considering only the individual galaxies in sub-region 
a when computing ^ as galaxy-scheme. 

In figure [S] we demonstrate this effect along with the 
effect of correlated sub-samples that was discussed before. 
The left-hand panel shows sub-sample estimates of the vari¬ 
ance of in a simulated survey (corresponding to setup I 
in table [TJ where the shape noise was put to zero and where 
400 sub-samples were used. Both the variance of C 
are severely underestimated on small scales, which is due 
to the correlation of sub-samples. At large angular scales, 
the galaxy-scheme yields systematically higher value for the 
variance than the pair-scheme and at least for it can 
even overestimate the variance. This is due to the missing 
cross-pairs in the re-sampling. 

For the right-hand panel of figure [5] we have generated 
a catalog of pure shape noise (ue. A, Ug^i as in setup I). This 
is the only situation where the assumption of uncorrelated 
sub-samples is valid. You can see that in this case the pair- 
scheme is able to estimate the variance without bias. The 
galaxy-scheme overestimates the variance for the reasons ex¬ 
plained before. A downside of the pair-scheme is that the 

^ a 

shear signals in the sub-measurements ^ become even more 
correlated, as can also be seen from the left-hand panel of 
figure [S] 


4.4 Jackknife 

Another me t hod of covariance estimation that 
iNorberg et al.l (l2009l ) investigate is the delete-one-jackknife. 
Instead of estimating the covariance of the measurements 
4 “ and rescaling it to the size of the whole survey the 
jackknife is considering the measurements 

_ E{i,j not in c.}('=t4 ± “ 0j\) 

^ ^ ^ notin ’ 

(32) 

i.e. the jackknife-sample a is generated by cutting out the 
subregion a and measuring the correlation functions in the 
rest of the survey. The j ackknife estimate for the covarian ce 
matrix is then given by llEfronlllQSl : iNorberg et al.ll2009l j 

N 

Cjnck = ^E(r“-r)^(r“-r), (33) 

q; = 1 

where ^ is the mean of all jackknife measurements. 

If we again assume that all subregions have the same 




































10 0. Friedrich, S. Seitz, T. F. Eifler, D. Gruen 




Figure 5. Different variance estimates using the sub-sample covariance estimator and 400 sub-samples. Left: Variance estimates for 4+ 
(solid lines) and (dotted lines) in a mock catalog without shape noise that is otherwise following setup 1. The red lines show the 
galaxy-scheme estimate (c.f. section l43t . the blue lines show the pair-scheme estimate and the black lines show the log-normal input 
model. Right: Sub-sample estimates of the variance of in a mock catalog that only consists of shape noise and has the same area 
and density as in setup I. It is only in this situation (and in the pair-scheme) that internal estimation of the covariance yields unbiased 
results. 


galaxy density and that the correlation functions are mea¬ 
sured on scales much smaller than the sub-region size then 
is approximately given by 


From this it also follows that 

j.*a. 1 


N 




V- 1 

V ■ I - r 

V- 1 

N-I-t 

N-1 

i-r 

N -1 ’ 


(N-l)- N 

V- 1 

(iV- 1) • V ^ 


EE^^ 


/5 77 ^^ 






(35) 


Inserting this into the definition of Cjack gives exactly the 
snbsample covariance Csc, i-e. on small angular scales the 
two methods are approximately equivalent 

In jackknife estimation one can in principle also differ¬ 
entiate between a pair scheme and a galaxy scheme. Using 
eq. [32] for corresponds to the galaxy scheme. This is 
equivalent to disregarding all pairs in the top panel of figure 


^ This is no general statement on the jackknife method. It holds 
only in our particular situation. 


[4] when computing The pair-scheme is given by dis¬ 
regarding all pairs in the lower left panel of figure [4] when 
computing In the pair scheme jackknife and sub-sample 
covariance become exactly equivalent when (assuming that 
each sub-patch has the same number of galaxies). 


4.5 Bootstrap Covariance 


The so called block bootstrap estimator of the covariance also 
divides the data into sub-samples. If the data is split into N 
sub-regions, then a number of Vboot bootstrap re-samplings 
of the data are generated by randomly drawing with replace¬ 
ment N of the sub- samples and combin i ng then into one re¬ 
sampled data s et jNor berg et al] l2009l : iNordman fc Lalii^ 
l2007l : lLohll2008l : lEfronll 198211 . If the correlation function mea¬ 
sured in the re-sampled data i {i = 1, ... , Vboot) is called 
^boot,i^ then the bootstrap estimate of the covariance is given 
by 


q -^boot 

\ ' /^boot.i ;rboot\T' / ^hoot.i jfo 

Cboot — ^ - i ) (? - i 

iVboot — 1 


(36) 

where ig now the mean of all 

Again, the question arises of whether one should con¬ 
sider the single galaxies or the galaxy pairs as the actual 
data (cf. section [4.311 . In what we will call galaxy-bootstrap 
one simply adds a copy of all galaxies in a sub-region a to 
the re-sampled data set i each time the sub-region a gets 
drawn. 

In the pair-bootstrap one adds all pairs associated to 
sub-region a to the list of pairs that is used to compute 
^boot.i difference between the two bootstrap schemes is 
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mainly the following: if the sub-region a gets drawn n times, 
then each pair in a gets a weight of n in the pair-scheme and 
a weight of in the galaxy-scheme. 

_ Note t hat the pair-bootstrap is very similar to what 

iLohl (l2008l 'l describes as marked point bootstrap, the only 
difference being, that we chose to split pairs between sub- 
regions evenly among these regions. 

We will see in section [5] that the galaxy-bootstrap 
severely overestimates the covariance. The other covariance 
estimators perform very similar to each other and suffer in 
similar ways from the systematics explained in subsection 
14.21 and 14.31 


4.6 Stability and Inversion of the Covariance 
Estimate 

All effects that bias the internal covariance estimate can in 
principle be minimized by dividing the data into very large 
sub-regions. This decreases both the correlation of the differ¬ 
ent sub-regions and the influence of pairs crossing between 
sub-regions. However, this also decreases the possible num¬ 
ber of re-samplings and hence increases the variance of the 
covariance estimator itself. 

In order to derive constraints on the number of re¬ 
samplings let us assume that we are in the limit were the cor¬ 
relations between sub-regions are small. Small here means 
that 

« (A^At;) , for a /3 . (37) 

As explained before, this is the only limit in which internal 
covariance estimation is valid. In this limit the sub-sample 
covariance is just a rescaling of the sample covariance of 
independent realizations of the sub-regions. Hence - in the 
limit considered here and under the assumption that the 
data vector behaves Gaussian - the sub-sample covariance 
estimate s are distributed a ccording to a Wishart distribu¬ 
tion fcf. iTavlor et al.ll2013l l. Also, the pair-jackknife is al¬ 
most equivalent to the pair-version of the sub-sample covari¬ 
ance, i.e. to eauation l28l when ^ is computed with equation 
ED Hence, also the pair-jackknife estimates should approxi¬ 
mately follow a Wishart distribution. 

The most important consequence of this is that the in¬ 
verse of the covariance matrix estimate will be a biased es¬ 
timate of the true inverse covariance matrix, and the bias 
is app roximately given by llHartlap et al.|[2007l : iTavlor et al.l 

l20ldh 

where N is the number of sub-regions and d is the number of 
data points in Especially, this factor has to be accounted 
for when computing the statistic, eq. 1241 i.e. it has an 
influence on the constraints derived on cosmological param- 
eter s when using intern al covariance estimation. 

ITavlor et al.l ll2013h also give constraints on N with re¬ 
spect to d when a certain accuracy in the final parameter 


constraints is demanded|3 We take their criterion, 

A > — -b (d -f 4) , (39) 

where e is the required fractional accuracy on parameter 
constraints, as a guideline also for internal covariance esti¬ 
mation. This is however under the assumption of an exact 
Wishart distribution, i.e. that the data vector is Gaussian 
and that the sub-regions are large enough to not cause sys¬ 
tematic biases in the covariance estimate. Demanding a frac¬ 
tional accuracy of e = 0.2 for the parameter constraints, this 
yields a necessary number of A > 54 -|- d re-sampling. Below 
this number there is no chance for internal covariance esti¬ 
mation to yield parameter constraints that are accurate to 
more that 20%. 


5 TESTING INTERNAL COVARIANCE 

ESTIMATORS ON SIMULATED COSMIC 

SHEAR SURVEYS 

We will now use the simulations described in section|3]to test 
the performance of internal covariance estimators. First, we 
will use setup I (cf. (D corresponding to a rather deep sur¬ 
vey. We carry out 50 independent realizations of this survey. 
In each survey we measure the correlation functions in the 
range and binning that was explained in section El We then 
estimate the covariance of the measured correlation func¬ 
tions using the different internal estimation schemes that 
were introduced in section E] Throughout this section - ex¬ 
cept for subsection ED- we consider the log-normal model 
that was explained in section 12.21 as the true covariance of 
the simulated surveys. This is justified by the fact that our 
results don’t change if we instead use the sample covariance 
of 1000 independent realisations that were presented in sec¬ 
tion El (cf. appendix [X] figure IA2II . 

In figure El we compare the sub-sample, jackknife and 
bootstrap estimates of the diagonal elements of the covari¬ 
ance matrix (both in the galaxy- and pair-scheme) when 
splitting the survey into N — 225 sub-regions. The most im¬ 
pressive finding is, that in the galaxy-scheme the bootstrap 
severely overestimat es the variance . This is in agreement 
with the findings of iNorberg et al] (l2009l i for galaxy clus¬ 
tering correlation functions. The duplication of whole sub¬ 
volumes of galaxies creates bootstrap samples that are in 
fact unrealistic, i.e. these bootstrap samples contain regions 
with no sources at all and on the other hand regions with 
a very high source density. Each original galaxy pair gets 
weighted by a factor of when the corresponding region is 
drawn n times. This puts a very high weight on small sub- 
areas of the bootstrap sample and creates an unphysically 
high variance between the bootstrap samples. 

In the pair-scheme however, all three internal estimators 
perform almost identical. This is not surprising, because in 
that scheme the bootstrap is just an approximation to the 
sub-sample covariance and sub-sample and jackknife covari¬ 
ance are almost identical in the pair-scheme. As explained in 
section m in the galaxy-jackknife scheme the two effects of 

® However, they are ignoring the impact that the variance in the 
inverted covarian ce estimate has on parame ter constraints, which 
is investigated bv ITavlor fc Joachimil ll2014lV 
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Figure 6. A comparison of the different internal estimation 
schemes when splitting the survey into N = 225 sub-regions. 
Green: galaxy-bootstrap, purple: pair-bootstrap, red: galaxy- 
jackknife, blue: pair-jackknife and cyan: sub-sample covariance 
compared to the analytical covariance (black line). We show the 
sub-sample covariance only in the galaxy-scheme because in the 
pair-scheme it is almost identical to jackknife and bootstrap. As 
explained in section |4] at large angular scales the different treat¬ 
ment of galaxy pairs crossing between sub-region leads to an over¬ 
estimation of the variance by the galaxy-scheme and an underes¬ 
timation of the variance by the pair-scheme. 


correlated sub-regions and false re-sampling of pairs partly 
cancel each other. Hence the galaxy-jackknife comes closest 
to the true variance at large scales. The performance of the 
sub-sample covariance (in the galaxy-scheme) only slightly 
differs from that. 

Because of the strong similarity between the different 
estimator we will restrict the following analyses to the pair- 
jackknife and the galaxy-jackknife. We now investigate the 
influence of sub-region size on internal covariance estima¬ 
tion. Hence we split the surveys into 3 different numbers of 
sub-regions: 10^, 15^ and 20^ corresponding to sub-region 
areas of approximately 7.0 x 7.0 deg^, 4.67 x 4.67 deg^ and 
3.5 X 3.5 deg^. In figure[3we compare the mean value of the 
50 jackknife estimates of the variance of (the diagonal el¬ 


ements of the covariance matrix) to the true underlying log¬ 
normal model. A comparison of the off-diagonal behaviour 
of the jackknife estimates to that of the input-covariance 
can be found in appendix]^ The errorbars in figure [7] repre¬ 
sent the standard deviation of the 50 jackknife estimates, i.e. 
they illustrate the noise of the internal estimators. You can 
see in this figure the biases in the jackknife estimates that 
we explained in the previous section. For ^+, both jackknife 
schemes underestimate the variance. At large scales, this is 
in the galaxy-jackknife scheme partly compensated by the 
false re-sampling of galaxy pairs. For the pair-jackknife 
underestimates the variance while the galaxy-jackknife over¬ 
estimates it. is a much more local measure in the sense 
that the different sub regions are less correlated in and 
that the covariance matrix is much more dominated by the 
shape noise contributions. Hence, the severe systematic un¬ 
derestimation of the variance that can be seen for does 
not appear as strongly for 

When increasing the number of sub-regions for the jack¬ 
knife estimators, the noise in the variance estimates becomes 
smaller but the deviations from the true variance also be¬ 
come stronger. This is because for smaller sub-regions the 

-'a 

estimated ^ become more correlated and because there will 
be more galaxy pairs crossing from one sub-region to an¬ 
other. 


5.1 Constraints on cosmological Parameters 

We will now take the 50 simulations as mock observations 
and try to constrain the dark matter density parameter ilm 
and the power spectrum normalization ag. To do so we 
sample the tlm-o's plane on a fine grid while keeping the 
other cosmological parameters fixed. Following a Bayesian 
approach we take the probability density in the parameter 
space to be proportional to the likelihood, 

p{tt) ~ £{tt) ~ exp , (40) 

where we assume our data vector ^ to be Gaussian such that 

= . (41) 

Here ^[tt] are our model predictions for (^) which we again 
compute with the nicaea package. We are assuming a prior 
of flm £ [0.1, 0.4] and as G [0.8,1.1], which is well centered 
around our input cosmology. For C we will either insert the 
log-normal model covariance or the jackknife estimates of 
the covariance. We will de-bias the inverse of the latter in the 
way explained in section lT6l Note that the reasoning in sec- 
tion l4.6l is in principle only valid for the pair-jackknife. And 
also for the pair-jackknife it is only valid in the case of almost 
uncorrelated sub-regions. We will nevertheless carry out the 
de-biasing in the same way for both jackknife schemes. Fur¬ 
thermore, we will al so ignore the variance o f the inverted 
covariance estimate (iTavlor fc Joachimill2014l ). as explained 
in the end of section [4] Our data vector ^ will be either 
or or the joint data vector of both correlation func¬ 
tions, in which case we will also take into account the cross 
covariance between the two. 

For each mock observation ^ a^nd for each avail¬ 
able covariance matrix we use equation Ho] to compute 
marginalised la constraints on Q.m and as, i.e. we consider 
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Figure 7. Mean values of 50 jackknife estimates of the variance of (left) and (right). Galaxy-jackknife was used for the red 
points while pair-jackknife was used for the blue points and the errorbars show the sample standard deviation of the single estimate (as 
estimated from the 50 jackknife matrices). The black line corresponds to the log-normal input model of the simulations. 
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the marginalised probability densities 

pni^m) = J das 

PaGs) = j dQ.„i p{^m,crs) (42) 

and we define la confidence interval to be that interval 
around the best fit parameter value which encloses ~ 68% of 
the probability and which has equal values of the probability 
density at each interval boundarj0. 

Because of the strong degeneracy between and as 
dKilbinger et alj l2013l : iKilbinger fc Schneider! l2004l l. even 
little uncertainties in the modelling of ^[tt H or in our simu¬ 
lations could shift the best-fit values of the parameters along 
the degeneracy. Fortunately, this does not affect our analy¬ 
sis because we only have to compare the constraints derived 
from the jackknife covariance estimates to the constraints 
obtained from the true (log-normal) covariance matrix. Fur¬ 
thermore, our results don’t change noticeably, if instead of 
the log-normal covariance matrix we use the sample covari¬ 
ance estimated from 1000 simulations (c.f. section[3l). Hence 
in any case, our analysis provides a fair test of internal co- 
variance estimators. 

In figure [8] we show the mean values of the upper and 
lower boundaries on and us as well as their mean best 
fit value for different numbers of jackknife re-samplings (red 
points and errorbars). The mean is taken with respect to 
all 50 confidence intervals we computed from the 50 mock 
observations. We also compare the jackknife constraints to 
those we get when using the true covariance matrix (blue 
lines). These figures only show the results for the galaxy- 
jackknife, which in the situation considered here yields the 
best agreement with the true covariance. 

We compare galaxy-jackknife and pair-jackknife in fig¬ 
ure [O] Here we show the mean width of the confidence inter¬ 
vals obtained with galaxy-jackknife, pair-jackknife and the 
true covariance matrix. For , the width of the confidence 
intervals agrees well with the confidence intervals obtained 
from the true covariance matrix. This is because the co- 
variance matrix of is dominated by its shape noise com¬ 
ponent, which is very accurately captured by jackknife. In 
fact, even for the pair-scheme and even for 400 jackknife 
re-samplings the width of the confidence intervals from 
alone is not underestimated. This seems to contradict figure 
[7] where the pair-scheme systematically underestimates the 
covariance. One reason for this is probably, that the variance 
in the inver ted covariance estimate i ncreases parameter un¬ 
certainties (iTavlor fc Joachimill201 j l. Note especially, that 
this is not the same effect as the de-biasing in eqn. 14.61 For 
^+, the strong underestimation of the covariance matrix by 
jackknife also leads to an underestimation of the uncertain¬ 
ties on and crs. Again one can see that the variance in 
the width of the confidence intervals (the errorbars in figure 
0 becomes smaller, when more jackknife re-samplings are 
used. In turn, this increases the overall underestimation of 
the uncertainties. If both correlation functions are combined 


® Without the last statement the definition of the la confidence 
interval would be ambiguous. 

In our modelling we are for example not considering the finite 
bin width in our measurement of 


and 225 re-samplings are used, the parameter uncertainties 
are underestimated by ~ 10%. 

We have not shown results from the pair-jackknife esti¬ 
mates in figure [8l but the best fit values of fim and as agree 
very well between the two jackknife schemes (i.e. within the 
green errorbars in figure 0, if only or are used to 
constrain the parameters. In figure fTOl we compare the pair- 
jackknife and galaxy-jackknife best fit values when using 
the full data vector. Here the pair-jackknife seems to yield 
a stronger bias of the best fit values with respect to the true 
covariance. 

The above results indicate that internal covariance es¬ 
timation can reproduce the constraints on parameters from 
the true covariance quite well, especially when the galaxy- 
jackknife scheme is used. However, these results are not gen- 
eralizable. In general, internal estimation of the covariance 
works best if the covariance matrix is shape noise dominated. 
Hence, the answer to what is the best estimation scheme and 
how well it can reproduce the true errorbars on cosmological 
parameters depends on the depth of the considered survey. 
A shallower survey not only has a smaller source density and 
hence a bigger shape noise. It also has a smaller convergence 
power spectrum which in turn reduces the cosmic variance 
part of the covariance. 

The procedure we presented above to investigate the 
performance of internal covariance estimators thus has to 
be re-run for each survey under consideration. One can con¬ 
sider the log-normal model as a good model for the true 
covariance of our simulations for mock catalogs with an 
area of > 1000 deg^ and a simple, connected geometry. For 
smaller surveys the finite-area-effect s hould not be ignored 
JSato et al.ll201l| : lKilbinger et al.ll2013h . However, these sur¬ 
veys can be simulated fast enough with our public code to 
generate a large sample of independent realisations of the 
mock data which provides a good sample covariance esti¬ 
mate of the true covariance matrix. This estimate can then 
be compared to an ensemble of internal covariance estimates 
as we have done it above. 

5.2 Matching the procedure to DES science 
verification and year 5 Data 

We will now present an application of our method. Our at¬ 
tempt is to determine the performance of internal covariance 
estimation for 

• setup Ila: Dark Energy Survey science verification data 
(DES-SV) 

• setup Ilb: DES year five data (DES-Y5) assuming a low 
source density 

• setup lie: DES year five data assuming a high source 
density. 

For the area, shape noise, source density and source redshift 
distribution cf. table [T] and section o A source density of 
10 arcmin~^ is forecasted for the final DES data while a 
density of ~ 6 arcmin“^ roughly corresponds to the current 
status of DES science verification data. Note also, that we 
are using a mask similar to the footprint of DES-SV to sim¬ 
ulate mock shape catalogs for setup Ha. Setups Hb and lie 
are simply simulated to be square shaped. _ 

W e adjust our data vector to that used bv iBecker et al.l 
ll2015ll . i.e. for both and we now use 15 logarithmic 
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Figure 8 . Mean Icr constraints on Q,rn and erg using galaxy- 
jackknife (red errorbars). The green errorbars show the standard 
deviation of the mean best-fit values (i.e. the standard deviation 
of the best fit values divided by . The blue lines indicate the 
constraints that are obtained when the true covariance is used 
in each mock catalog. Note that the error bars are very symmet¬ 
ric. For surveys as big as our simulations the constraining power 
becomes large enough to turn the - usually banana shaped - de¬ 
generacy between Qm and as into almost elliptical contours in 
the parameter plane (c.f. appendix [bI 


Figure 9. Mean width of the la uncertainty on Qrn and as us¬ 
ing pair-jackknife (blue) and galaxy-jackknife (red). The error- 
bars show the standard deviation the 50 estimated confidence 
intervals. The black dotted line indicates the mean width of the 
confidence intervals when the true covariance is used in each mock 
catalog. 


bins ranging from 0 = 2 aremin to ^ = 300 aremin. We will 
cut the survey into 100 sub-regions for setup Ila. Note that 
this way our biggest angular scales by far exceed the diame¬ 
ter of our subregions which is ~ 45 aremin. Hence, this can 
be considered an on-the-edge test of internal covariance esti¬ 
mators. A good tool to define sub-regions in an irregular sur- 
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Figure 10. Mean best fit values of Qrn and erg using pair- 
jackknife (blue) and galaxy-jackknife (red). The errorbars show 
the standard deviation of the mean, as estimated from the 50 best 
fit values. The black dotted line indicates the mean best fit value 
when the true covariance is used in each mock catalog. 


vey geometry is the kmeans algorithiiF^. For setups lib and 
lie we decide to split the survey into 225 sub-regions which 
corresponds to a diameter of ~ 4.7 aremin. This should give 
a more stable estimate of the covariance while still yielding 
much larger sub-regions than in setup Ila. 

In figure [11] we compare the internal variance estimates 
to the true covariance. The latter is taken to be the log¬ 
normal model for the Y5 simulations and a sample vari¬ 
ance computed from 1000 independent realisations for the 
SV simulations. Because of the fewer number of bins we 
are now using the procedure described in section 12.2.31 to 
compute the log-normal covariance matrix. As you can see, 
for G the pair-jackknife now becomes the best estimator 
of the variance. For the situation is similar to what we 
have seen before, i.e. both schemes mostly underestimate 
the variance and the galaxy-jackknife is overall closer to the 
true variance. Hence, judging from figure [TT] we conclude 
that galaxy-jackknife should be used in order to not under 

implemented by Erin Sheldon for python, 
www.github.com/esheldon/kmeans_radec 


estimate the true uncertainties in the data vector. However, 
these statements only hold for the diagonal elements of the 
covariance matrix. A convenient way to compare the com¬ 
plete covariance estimates is to derive likelihood contours 
from them in the desired parameter space. 

We carry out a likelihood analysis in the flm-crs plane 
for the 10 simulations that have a Y5-like area and a 
source density of 6 arcmin“^ which is the highest den- 
sity currently achi eved in DES science verification data 
ll Becker et al.l l2015l ). In figure [T^j we show the likelihood 
contours obtained from one of the simulations when using 
galaxy-jackknife, pair-jackknife and the log-normal model 
for the covariance matrix. The contours were obtained from 
Monte-Carlo-Markov- Chains (150.000 step s) using the COS- 
MOLIKE package by lEifler et al.l ll2014l ). We present the 
likelihood contours from the other 9 independent simulations 
in appendix |B| As expected, jackknife estimation underes¬ 
timates the uncertainties. The input cosmology lies within 
the 1-(T contour in 6 of 10 simulation, when the log-normal 
covariance is used. It lies within the l-cr contour in 5 of 10 
simulation, when the covariance is estimated with jackknife 
(either scheme). 

In table[5]we show the average ratio of the volume in the 
flm-cTs plane enclosed by the la- and 2(T-contours when us¬ 
ing jackknife to that when using the true covariance matrix. 
Since the la- and 2(T-ellipses obtained from jackknife and 
from the true covariance lie well on top of each other, this ra¬ 
tio can be considered as the fraction of the true uncertainties 
that is recovered by the jackknife covariance matrices. You 
can see from table[2]that the volume inside contours of con¬ 
stant likelihood in the firn-ag plane estimated with galaxy- 
jackknife is on average > 85% of the true volume while the 
volume estimated with pair-jackknife recovers only > 70% 
of the true volume. This agrees with the impression (from 
figures [Bl] and [B^ that the contours obtained with galaxy- 
jackknife match better to the contours obtained from the 
true covariance. Note also, that the ellipses obtained from 
pair-jackknife have in some cases a strong off-set along the 
degeneracy between and cts compared to the true covari¬ 
ance and the galaxy-jackknife estimates. This is probably 
because pair-jackknife strongly underestimates the variance 
of at large angular scales, which causes even small fluc¬ 
tuations at these scales to shift the contours considerably. 

Finally, we want to see how well jackknife matrices re¬ 
cover the uncertainties perpendicular to the degeneracy be¬ 
tween flrn and ag. To do so, we consider the parameter com¬ 
bination 


Eg := 


as 

0.9 



0.5 


(43) 


Contours of constant Eg are roughly parallel to the degen¬ 
eracy that can be seen in figures 1121IBII and IB2I For each 
of our 10 realisations we bin our MCMC’s in Eg to estimate 
its probability density. Table [3] displays the average ratio of 
the Icr and 2a uncertainties obtained from jackknife to the 
uncertainties obtained from the true covariance. This time, 
we find that galaxy-jackknife on average yields ~ 90% of the 
true uncertainties while pair-jackknife yields ~ 85%. Hence, 
when the degeneracy between fl„, and erg is broken by other 
probes (such as the power spectrum of temperature fluctua- 
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Figure 11. Variance estimates for DES-SV like data (left), DES-Y5 like data with a low density (middle) and with a high density 
(right). Red dots show the galaxy-jackknife estimates and blue dots the pair-jackknife estimates. For the Y5 case the lognormal model 
together with egn. IlSI was taken as a reference covariance (black lines) while for the SV case we estimated the true covariance from 1000 
independent realizations of the mock data in order to account for the finite-area-effect. The errorbars indicate the standard deviation of 
the single estimates as obtained from 10 independent measurements. 


tion in the cosmic microwave background) the performance 
of jackknife covariance matrices slightly improves. 

Judging from the above numbers and from the contours 
in appendixl^we deem that > 85% of the true uncertainties 
on Qrn and as in a 2D cosmic shear analysis can be recov¬ 
ered without the use of large suits of N-body simulations 
or covariance models. When other probes like the CMB are 
used to break the degeneracy between the two parameters, 
the performance of jackknife even increases, because the de¬ 
viations from the true covariance mostly take place along 
the direction of degeneracy between Qrn and os. 


6 CONCLUSIONS 

We have explored the performance of internal covariance es¬ 
timation for cosmic shear 2-pt. correlation functions. We 
devised two different jackknife schemes and explained in 
detail when these schemes underestimate the true covari¬ 
ance and when overestimation takes place. Furthermore, 
we explained why the sub-sample covariance and the pair- 
bootstrap covariance yield results that are very similar to 
jackknife estimation of the covariance matrix. Based on the 
pair-jackknife sche me we have argued that the Ander son- 
Hartlap-Kaufman llKaufmanlll967l : iHartlan et al.ll2007l) de¬ 
biasing factor should also be applied when inverting jack¬ 
knife covariance matrices. Based on empirical findings we 
also recommend the use of this factor for the galaxy- 
jackknife scheme. 

We have demonstrated our findings in an exemplary 


study using log-normal simulations of the convergence field 
and the corresponding shear field. We found the performance 
of all internal covariance estimators - except for the boot¬ 
strapping of galaxies - to be very similar. For the inves¬ 
tigated cases, jackknife covariance matrices could provide 
accurate uncertainties on cosmological parameters as com¬ 
pared to the true covariance matrix of our simulations. Our 
conclusions regarding the two possible re-sampling schemes 
are the following: 

• galaxy-bootstrap severely overestimates the covariance, 
which is in agreement with the finding of iNorberg et al.l 
(l2009l) for galaxy clustering correlation functions. 

• from alone, the pair-jackknife scheme reconstructs 
the parameter constraints most faithfully (cf. figure [S|. 

• from ^-1- alone and when combining the two correlation 
functions, we find that the parameter constraints are best 
reconstructed by the galaxy-jackknife. 

The performance of the galaxy-scheme turns out to be bet¬ 
ter in most situations, because two systematic errors (cf. 
sections 1121 and B21) cancel each other partly in the that 
scheme. The pair-jackknife suffers from only one of these 
systematics and hence always yields lower (absolute) val¬ 
ues for the covariance than the galaxy-jackknife and always 
underestimates the (absolute) values of the true covariance 
matrix. 

Our results can not be generalized to arbitrary surveys, 
i.e. our paper rather demonstrates a general method to find a 
good covariance estimation scheme for any particular survey. 
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Figure 12. l-tr and 2-cr contours in the Qrn-o'S plane obtained 
from the two jackknife schemes (read and blue) and the true co- 
variance (log-normal covariance, black) and using the combined 
data vector (5+, ^_). The input cosmology lies within the l-cr con¬ 
tour in 6 of 10 simulation, when the log-normal covariance is used. 
It lies within the l-cr contour in 5 of 10 simulation, when the co- 
variance is estimated with jackknife (either scheme). In appendix 
[3we show the contours obtained from the other simulations. The 
underestimation of the uncertainties by jackknife mainly takes 
place along the direction of the degeneracy between Clrn and erg. 

In making our simulation code public we provide our readers 
with a tool to re-do the presented analyses for their desired 
set-up. As an application example we tested jackknife esti¬ 
mation of the covariance for a 2D cosmic shear analysis of 
the Dark Energy Survey. We found that for the complete, 
5-year DES survey internal covariance estimators can pro¬ 
vide reliable parameter constraints in a 2D cosmic shear 
analysis. We recommend a scheme of ~ 15 x 15 jackknife 
re-samplings to yield a stable covariance matrix. Judging 
from Hgures 1121IBII and IB2I we find as before that the like¬ 
lihood contours in the Qm-o's plane are best reconstructed 
by the galaxy-jackknife scheme, if both correlation functions 
and are combined. This way, on average > 85% of the 
true uncertainties are captured by the internally estimated 
covariance matrix. If the degeneracy between Dm, and ag is 
broken, this value increases to ~ 90%. Hence, up to ~ 90% 
of the true uncertainties in a 2D cosmic shear analysis can 
be provided from internally estimated covariance matrices. 


galaxy-jackknife pair-jackknife 

Em.jack/Via,true 0.86 ± 0.08 0.72 ± 0.09 

V2,,,jack/V2a,true 0.87 ± 0.08 0.74 ± 0.09 

Table 2. Ratio of the volume within the Icr and 2cr contours in 
the Qrn — o-g plane obtained from jackknife and true covariance 
(setup lib). The errors are given by the standard deviation of a 
sample of 10 independent simulations. The combined data vector 
of and was used. 

galaxy-jackknife pair-jackknife 

ASg l,,,jack/AS8 i,^,true 0.91 ± 0.08 0.86 ± 0.10 

ASg 2o-.jack/AS8 2CT.true 0.90 ± 0.08 0.85 ± 0.09 

Table 3. Ratio of the Itr and 2cr uncertainties on Eg ~ o-gflm® ob¬ 
tained from jackknife and true covariance (setup lib). The errors 
are given by the standard deviation of a sample of 10 independent 
simulations. Again, the combined data vector of 5_|_ and was 
used. 
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APPENDIX A: CORRELATION MATRICES 
AND CONSTRAINTS FROM EMPIRICAL 
COVARIANCE 


To see how jackknife estimates of the covariance matrix cap¬ 
ture the cross-correlations between different bins of xi± we 
are looking at the correlation matrix. This matrix is given 
in terms of the covariance matrix elements Cavij as 


CoTTij 


COYi 


\J COYiiCOYj 


(Al) 


In the top panel of hgure lATl we compare the correlation ma¬ 
trix obtained from the log-normal model covariance to the 
correlation matrix obtained from averaging all 50 jackknife 
estimates of the covariance matrix that were presented in 
section [5l i.e. using setup I from table [T] (we show here the 
jackknife with 400 re-samplings which divided the survey 
into the smallest sub-regions). The lower left corner shows 
the auto correlations of and the upper right corner shows 
the auto correlations of The upper left and lower right 
corners show the cross-correlations between the two correla¬ 
tion functions. Furthermore, the lower right half of the plot 
shows the correlations obtained from the log-normal model 
and the upper left half shows the correlations obtained from 
the average jackknife covariance estimate. Each column and 
row of pixels represents one angular bin and the bins range 
from 1' to 150', starting on the lower left corner. 

The top panel of figure ED indicates, that jackknife is 
able to capture the general structure of the correlation ma¬ 
trix of the 2-pt correlation functions. Given that internal co- 
variance estimators mostly underestimate the variance of 
one can hence conclude that the covariance elements Covij 
are approximately underestimated by the same amount as 
the square root of CoYuCoYjj (cf. eon. lAll) . 

In the bottom panel of figure ED we show the same 
plot but using the empirical covariance matrix obtained from 
1000 independent realizations of setup I in the lower right 
half of the plot. The empirical covariance matrix is obvi¬ 
ously noisier than the log-normal model covariance matrix. 
In order to confirm, that there are nevertheless no signifi¬ 
cant deviations of our simulations from the log-normal input 
model, we show in figure[A2]again the constraints on and 
<78 that where presented in figure [8l but this time using the 
empirical covariance to compute the reference constraints. 




Figure Al. Correlation matrix of (^j-. Bins 0 to 34 belong to 
and bins 35 to 69 belong to For each 2-pt function the bins 
range from 1' to 150', starting on the lower left corner. Top panel: 
the lower right half of the plot displays the correlation coefficients 
of obtained from the log-normal model and the upper left 
halt displays the correlation coefficients obtained from the average 
jackknife covariance estimate (using 400 re-samplings). Bottom 
panel: same plot, but this time the empirical covariance from 1000 
realizations of setup I. Note that we are averaging over 50 different 
jackknife estimates of the covariance matrix here, which makes 
the empirical covariance matrix seem noisier than the jackknife 


covariance. 
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APPENDIX B: LIKELIHOOD CONTOURS 

Figures |BT] and |B^ show the 1- and 2-cr contours in the Qm- 
ag plane computed with COSMOLIKE when using galaxy- 
jackknife and pair-jackknife to estimate the covariance ma¬ 
trix (red and blue lines) and compare them to the same 
contours obtained from the true covariance matrix (black 
lines). The simulations are configured to mimic the com¬ 
plete, 5 year Dark Energy Survey (cf. section 15.21 or table 
[T] setup Ilb). The only thing that differs from simulation 
to simulation is the random seed that was used to generate 
the log-normal fields and the shape noise. The green dots 
represent the input cosmology of the simulations. 
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Figure A2. These plots are identical to the ones shown in figure 
[S] except that this time the blue lines indicate the mean best fit 
values and 1 — a constraints obtained from the empirical covari¬ 
ance matrix, i.e. from the sample covariance of 1000 independent 
realizations of setup 1. 
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Figure Bl. 1- and 2-cr contours in the Hm-crs plane obtained from the first 6 simulations. 
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Figure B2. 1- and 2-a contours in the Hm-crs plane obtained from the remaining 4 simulations. 





